Testing different image augmentation

The following runs are used to evaluate which model configuration perform better. The average F1 and F2 scores of the 10 runs are used as the evaluation metric.

The ten runs have different training datasets and testing datasets. The idea of splitting the run into 10 runs is because I want to see how the model configuration performs on different testing dataset. The idea is similar to k-fold cross validation, where each fold has trained on the different training dataset and validated on the different validation datasets. The reason why I train the model this way instead of the actual k-fold cross-validation is that I want to make sure that the testing datasets consist of images that the model has never seen before (images from different videos) since multiple frames from the same video are similar to each other, and those frames can be ended up in testing and validation datasets resulting in a high validation score.

The ten runs are used to evaluate the different model configuration. The actual model for each model configuration is a model trained and validated using the all dataset.

For the model I used in the fine-tuning, I created a new model that used all datasets (including the testing dataset) in training and validation. For the DVN resnet 50 model I used in the fine-tuning with SD1, I created a model using all images from the dataset.

Deven commented that we cannot evaluate this way because there was no control. The test dataset will need to be the same.

Sharpen vs No Sharpen runs

The baseline model's config:

• Backbone: ResNet50
• Image argumentation: Resize((470, 700),method='squish', pad_mode='zeros')
• Batch Size: 32
• Epoch: 10
• Metric threshold = 0.5

The sharpen model's config:

• Backbone: ResNet50
• Image argumentation: '[Resize((470, 700),method='squish', pad_mode='zeros'), SharpenTransform(50)]
• Batch Size: 32
• Epoch: 10
• Metric threshold = 0.5

The table compares the performance of baseline model to the sharpen model. The comparison is done across the column. Each row represents the results of the two models trained on one training dataset and tested on one testing dataset.

Train data	F1 normal		F1 average (no ND)		F2 average (no ND)
	baseline	sharpen	baseline	sharpen	baseline	sharpen	Label in test dataset
39	0.9626	0.966	0.5514	0.4584	0.5323	0.4311	['ND', 'TB', 'TBA', 'AMH']
40	0.8776	0.8833	0.2901	0.3449	0.2554	0.3221	['ND', 'TB', 'TBA', 'AMH']
41	0.9708	0.9721	0.1269	0.1532	0.1178	0.1471	['ND', 'TB', 'TBB', 'TBA', 'AMH']
42	1	0.9864	N/A	N/A	N/A	N/A	['ND']
43	0.9881	0.9857	0.9136	0.6064	0.8836	0.593	['ND', 'TF', 'TB', 'AMH']
44	0.9979	0.9867	0.5668	0.4655	0.5464	0.4561	['ND', 'TF', 'AMH']
45	0.8938	0.9127	0.3442	0.4136	0.3485	0.4354	['ND', 'TBA', 'TB', 'AMH', 'RFJ']
46	0.9361	0.9402	0.5451	0.6185	0.4783	0.5604	['ND', 'DAGS', 'TB', 'AMH']
47	0.9576	0.9371	0.6841	0.6222	0.651	0.5687	['ND', 'TB', 'DAGS', 'AMH']
48	0.9712	0.9489	0.2861	0.2545	0.2708	0.26	['ND', 'TB', 'AMH', 'DAGS', 'TBB', 'SSS', 'TBA']
Average	0.95557	0.95191	0.4787	0.437466667	0.453788889	0.419

No sharpen: Threshold = 0.50 vs Threshold = 0.30

Run ID	F1 normal		F1 average (no ND)		F2 average (no ND)
	thres=0.5	thres=0.3	thres=0.5	thres=0.3	thres=0.5	thres=0.3	Label in test dataset
39	0.9626	0.9576	0.5514	0.5493	0.5323	0.537	['ND', 'TB', 'TBA', 'AMH']
40	0.8776	0.8639	0.2901	0.3292	0.2554	0.3041	['ND', 'TB', 'TBA', 'AMH']
41	0.9708	0.9671	0.1269	0.1279	0.1178	0.1189	['ND', 'TB', 'TBB', 'TBA', 'AMH']
42	1	1	N/A	N/A	N/A	N/A	['ND']
43	0.9881	0.9853	0.9136	0.9514	0.8836	0.9549	['ND', 'TF', 'TB', 'AMH']
44	0.9979	0.9969	0.5668	0.7046	0.5464	0.6549	['ND', 'TF', 'AMH']
45	0.8938	0.8934	0.3442	0.3302	0.3485	0.3418	['ND', 'TBA', 'TB', 'AMH', 'RFJ']
46	0.9361	0.9331	0.5451	0.5651	0.4783	0.5098	['ND', 'DAGS', 'TB', 'AMH']
47	0.9576	0.96	0.6841	0.6709	0.651	0.6574	['ND', 'TB', 'DAGS', 'AMH']
48	0.9712	0.9635	0.2861	0.284	0.2708	0.2738	['ND', 'TB', 'AMH', 'DAGS', 'TBB', 'SSS', 'TBA']
Average	0.95557	0.95208	0.4787	0.5014	0.453788889	0.483622222

Sharpen: Threshold = 0.50 vs Threshold = 0.30

Run ID	F1 normal		F1 average (no ND)		F2 average (no ND)
	thres=0.5	thres=0.3	thres=0.5	thres=0.3	thres=0.5	thres=0.3	Label in test dataset
39	0.966	0.9628	0.4584	0.4705	0.4311	0.4516	['ND', 'TB', 'TBA', 'AMH']
40	0.8833	0.867	0.3449	0.3624	0.3221	0.345	['ND', 'TB', 'TBA', 'AMH']
41	0.9721	0.9716	0.1532	0.1534	0.1471	0.1486	['ND', 'TB', 'TBB', 'TBA', 'AMH']
42	0.9864	0.9898	N/A	N/A	N/A	N/A	['ND']
43	0.9857	0.9825	0.6064	0.7335	0.593	0.7002	['ND', 'TF', 'TB', 'AMH']
44	0.9867	0.9857	0.4655	0.4746	0.4561	0.4698	['ND', 'TF', 'AMH']
45	0.9127	0.9102	0.4136	0.4076	0.4354	0.4428	['ND', 'TBA', 'TB', 'AMH', 'RFJ']
46	0.9402	0.9354	0.6185	0.6435	0.5604	0.5815	['ND', 'DAGS', 'TB', 'AMH']
47	0.9371	0.9367	0.6222	0.6594	0.5687	0.6084	['ND', 'TB', 'DAGS', 'AMH']
48	0.9489	0.9456	0.2545	0.2546	0.26	0.2656	['ND', 'TB', 'AMH', 'DAGS', 'TBB', 'SSS', 'TBA']
Average	0.95191	0.94873	0.437466667	0.462166667	0.419322222	0.445944444

No sharpen: No additional image transforms vs Additional image transforms (Flip(), Brightness(), Warp(), Rotate())

Run ID	F1 normal		F1 average (no ND)		F2 average (no ND)
	no additional image transform	addition image transforms	no additional image transform	addition image transforms	no additional image transform	addition image transforms	Label in test dataset
39	0.9626	0.9723	0.5514	0.4819	0.5323	0.5323	['ND', 'TB', 'TBA', 'AMH']
40	0.8776	0.8849	0.2901	0.3665	0.2554	0.356	['ND', 'TB', 'TBA', 'AMH']
41	0.9708	0.971	0.1269	0.1463	0.1178	0.141	['ND', 'TB', 'TBB', 'TBA', 'AMH']
42	1	0.9932	N/A	N/A	N/A	N/A	['ND']
43	0.9881	0.9933	0.9136	0.762	0.8836	0.7281	['ND', 'TF', 'TB', 'AMH']
44	0.9979	0.9979	0.5668	0.9918	0.5464	0.9967	['ND', 'TF', 'AMH']
45	0.8938	0.912	0.3442	0.4076	0.3485	0.4428	['ND', 'TBA', 'TB', 'AMH', 'RFJ']
46	0.9361	0.9325	0.5451	0.5829	0.4783	0.5293	['ND', 'DAGS', 'TB', 'AMH']
47	0.9576	0.9402	0.6841	0.6378	0.651	0.6076	['ND', 'TB', 'DAGS', 'AMH']
48	0.9712	0.9575	0.2861	0.2614	0.2708	0.2483	['ND', 'TB', 'AMH', 'DAGS', 'TBB', 'SSS', 'TBA']
Average	0.95557	0.95548	0.4787	0.515355556	0.453788889	0.509122222

Sharpen: No additional image transforms vs Additional image transforms (Flip(), Brightness(), Warp(), Rotate())

Run ID	F1 normal		F1 average (no ND)		F2 average (no ND)
	no additional image transform	addition image transforms	no additional image transform	addition image transforms	no additional image transform	addition image transforms	Label in test dataset
39	0.966	0.9427	0.4584	0.4578	0.4311	0.4223	['ND', 'TB', 'TBA', 'AMH']
40	0.8833	0.9131	0.3449	0.3916	0.3221	0.3683	['ND', 'TB', 'TBA', 'AMH']
41	0.9721	0.9718	0.1532	0.1547	0.1471	0.1506	['ND', 'TB', 'TBB', 'TBA', 'AMH']
42	0.9864	0.9864	N/A	N/A	N/A	N/A	['ND']
43	0.9857	0.9899	0.6064	0.9234	0.593	0.8983	['ND', 'TF', 'TB', 'AMH']
44	0.9867	0.9867	0.4655	0.4655	0.4561	0.4561	['ND', 'TF', 'AMH']
45	0.9127	0.9127	0.4136	0.4136	0.4354	0.4354	['ND', 'TBA', 'TB', 'AMH', 'RFJ']
46	0.9402	0.9402	0.6185	0.6185	0.5604	0.5604	['ND', 'DAGS', 'TB', 'AMH']
47	0.9371	0.9371	0.6222	0.6222	0.5687	0.5687	['ND', 'TB', 'DAGS', 'AMH']
48	0.9489	0.9489	0.2545	0.2545	0.26	0.26	['ND', 'TB', 'AMH', 'DAGS', 'TBB', 'SSS', 'TBA']
Average	0.95191	0.95295	0.437466667	0.477977778	0.419322222	0.457788889

Colour vs Black&White

• The following image augmentations were applied to
  • colour runs: Resize((470, 700),method='squish', pad_mode='zeros'); [Flip(),Brightness(), Warp(), Rotate()]
  • BW runs: ImageBlock(cls=PILImageBW);Resize((470, 700),method='squish', pad_mode='zeros'); [Flip(),Brightness(), Warp(), Rotate()]

Run ID	F1 normal		F1 average (no ND)		F2 average (no ND)		Label in test dataset
	colour	BW	colour	BW	colour	BW
39	0.972	0.961	0.532	0.561	0.532	0.540	['ND', 'TB', 'TBA', 'AMH']
40	0.885	0.843	0.356	0.243	0.255	0.216	['ND', 'TB', 'TBA', 'AMH']
41	0.971	0.968	0.141	0.157	0.118	0.139	['ND', 'TB', 'TBB', 'TBA', 'AMH']
42	0.993	0.993	N/A		N/A		['ND']
43	0.993	0.984	0.728	0.583	0.884	0.572	['ND', 'TF', 'TB', 'AMH']
44	0.998	0.998	0.997	0.492	0.546	0.497	['ND', 'TF', 'AMH']
45	0.912	0.910	0.443	0.363	0.349	0.361	['ND', 'TBA', 'TB', 'AMH', 'RFJ']
46	0.933	0.930	0.529	0.507	0.478	0.456	['ND', 'DAGS', 'TB', 'AMH']
47	0.940	0.940	0.608	0.681	0.651	0.648	['ND', 'TB', 'DAGS', 'AMH']
48	0.958	0.958	0.248	0.243	0.271	0.224	['ND', 'TB', 'AMH', 'DAGS', 'TBB', 'SSS', 'TBA']
Average	0.955	0.948	0.509	0.426	0.454	0.406